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WO 98/49631 PCT/IB98/00594 
System for composing multimedia letters. 



BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The invention relates to a system for composing and sending multimedia 

e-mails. 

5 

2. Related Art 

There are a number of systems for creating multimedia e-mail in a 
computer work station or personal computer environment. These include Alta Vista's 
"HOWDY"; EUDORA mail; Sun's SOLARIS e-mail package; and Pineapple's 

10 "ANIMATED GREETINGS". 

These systems suffer from several drawbacks. First, incompatibilities 
between various machines and e-mail software packages in a network, such as the Internet, 
often lead to the format of the multimedia message being undecipherable by the receiving 
system. Second, if the recipient has a relatively simple piece of equipment, which functions 

15 only as a web browser, the recipient may not be able to receive a complex multi-media e- 
mail. Third, the process of composing multimedia e-mail on the sender's end is very 
cumbersome, including 1) having to run separate programs to capture/acquire content like 
video, audio, and images; 2) having to save the content into individual files on local disk; 
and 3) specifying those filenames to the e-mail composition software. 

20 

SUMMARY OF THE INVENTION 

The object of the invention is to create a system which can allow the 
recipient to receive multimedia e-mails with personal content of the sender while avoiding the 
listed drawbacks. 

25 This object is achieved by a system in which e-mails are stored on a 

central server so that they can be browsed by low function equipment so long as the user 
knows the necessary security information. 

This system has the additional advantage that the e-mail facility provides a 
draw to the central server. If the central server provides advertising or infomercial type 
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information, users will be exposed to such information in making use of the e-mail facility. 

Some systems which allow users to compose postcards on a server are 
known. Examples are shown at 

http : //www. yahoo . com/Entertainment/Humor_Jokes_and_Fu n/Gr eeting_Cards_on_www/ . 
5 Such systems are not true e-mail systems, because there is no personal multimedia content. 
The users are constrained to use canned multimedia content from the web site. 



BRIEF DESCRIPTION OF THE DRAWING 

The invention is described by way of non-limitative example with 
10 reference to the following figures. 

Fig. 1 shows a network in which the invention can operate. 
Fig. 2 shows a user system. 

Fig. 3a shows a flow chart of the operation of the invention from the 
point of view of the server. 
15 Figs. 3b is a flowchart of the plug-in DLL and further expands box 307 

of fig. 3a from the point of view of the user system or browser. 

Fig. 3c is a flowchart of the asset capture DLL. 

Fig. 4 shows more aspects of the user system. 

Figs. 5a-f show screen appearances of a preferred embodiment of the 

20 invention. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Fig. 1 shows a system which includes a network 101, such as the internet, 
a server 102, one user's system 103, and another user's system 104. For the purpose of this 

25 disclosure, the one user is going to be the sender of the multimedia e-mail message, while 
the other user is going to be the receiver of the multimedia e-mail message. 

Fig. 2 shows more detail of the one user's system 103. The system 
includes a network interface unit 201. A network interface unit can be any standard unit, 
such as a telephone modem, a cable modem, or an ethernet card. The system also includes a 

30 display device 202, which can be a TV monitor, computer monitor, LCD screen or other 
display. The composition unit can be any processor such as a PC or set top box. There is 
a user input device 204 such as a wired .keyboard, a wireless keyboard, a touch sensitive 
component of the display, a telephone keypad, a TV remote control, or a voice recognition 
unit. Content for a multimedia e-mail message is to be acquired from content sources, such 
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as a VCR 205, a video camera 206, a microphone or other audio equipment 207, a PC hard 
drive or CD-ROM 208, or a scanner 209. 

Fig. 3a shows a detailed flow chart of the operation of the system 
according to the invention from the point of view of the server. 
5 At 301 the user, who wishes to create a multimedia e-mail, visits the web 

site. At 302, the user logs in or registers and is assigned a session i.d. and, if a new user, a 
user i.d. At 303, the user is shown an HTML page with command options. Fig. 5a shows 
the page of the preferred embodiment. 

At 315, it is tested whether a send command is received. If not, the 

10 system receives a user asset selection at 304. Fig. 5b shows an asset selection page of the 
preferred embodiment. The selection is achieved by the user by clicking on a button on the 
user screen. This screen includes 4 buttons for text 501, audio 502, still photos 503, or 
video 504, respectively. 

The system then retrieves an asset "plug-in" for the type of asset selected. 

15 "Plug-ins** are standard extensions to web browsers. Here plug-ins are used to capture 
individual types of assets. The code is particular to the type of device which captures the 
asset. For example, the plug-in to scan and digitize a photograph will use the industry 
standard TWAIN scanner library and specifications to interact with any TWAIN compliant 
scanning device. The plug-in for capturing audio and video will use standard Microsoft 

20 libraries, like MFC and /or Video for Window, on a PC. The plug-ins are preferably 
implemented as DLL's (Dynamic Link Libraries) according to the plug-in specification 
published by Netscape. 

More information about plug-ins can be found at 
http://home. netscape, com/eng/mozilla/3. 0/handbook/pgguide. htm. 

25 Other plug-ins can be found associated with Netscape Navigator and the Microsoft Internet 
Explorer. Other platforms, for instance set top boxes or digital TVS, have software 
development tools which allow those of ordinary skill in the art to develop equivalents of 
plug-ins for their own platforms. Accordingly, the architecture shown herein is readily 
portable to all Internet browsers. 

30 Figs. 3b & c show two portions of the operation of box 307 from the 

point of view of the browser. The portion of Fig. 3b is the portion used by the browser to 
receive assets and send the captured assets to the server. Fig. 3b is known as the plug-in 
DLL. The software of Fig. 3b also creates interface screens for the user. The portion of 
Fig. 3c shows the software which interacts with the driver software and acquisition 
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hardware, e.g. the video capture board, to capture the asset and save it on the local disk. 
The module of Fig. 3c is also called the asset capture DLL. The functions of Fig. 3b and 3c 
could be combined by one of ordinary skill in the art to create a monolithic piece of software 
which performs both functions. 

5 Fig. 3b starts at 320 with the reception of parameters from the browser: 

session i.d.; user i.d.; window size; colors; etc. At 321 and 322 the plug-in DLL, used in 
Fig. 3b, and the asset capture DLL, used in Fig. 3c, are initialized. At 323, the user screen 
is created or modified. Examples of user screens are shown in Figs. 5a-f. At 324, the 
software waits for user commands or messages from the asset capture DLL. 

10 At 325 the software determines whether it is time to start capture. If so, 

at 326, the asset capture DLL is requested to capture the asset and control is returned to 323. 

If it is not time to start capture, at 327, it is tested whether it is time to 
stop capture. If so, the asset capture DLL is requested to stop capture at 328 and control is 
15 returned to 323. 

If it is not time to stop capture, at 329 it is tested whether it is time to 
play a captured asset. Such play would be in response to a user request. If there has been 
such a request, at 330, the asset capture DLL is requested to play the recorded asset. If 
there has not been such a request, then control proceeds to 331. 
20 At 331, it is tested whether it is time to send a captured asset to the 

server. If so, at 332, the filename of the captured asset is provided to the browser and the 
browser is asked to upload the file to the server. Thereafter, at 334, the plug-in DLL is 
deinitialized at 334 and this procedure ends. 

If, at 331, it is not time to send a captured asset to the server, some error 
25 has occurred. Accordingly, a time out notification from the asset capture DLL is sent at 333 
and a diagnostic message is displayed for the user at 335. Then control is returned to 323. 

Fig. 3c shows operation of the asset capture DLL. At 340, the asset 
capture DLL is initialized. Then at 341, the asset capture DLL waits from a request from 
the plug-in DLL, an error, or a timeout. At 342, it is tested whether there is a capture 
30 request from the plug-in DLL. If so, a max timeout time is set at 352. The capture device 
is requested to start capture at 351, and control returns to 341. 

If there is no request to start capture, then it is tested at 343 whether there 
is a request to stop capture. If there is a request to stop capture, then the asset is saved to a 
file at 344, the plug-in DLL is notified of the file name, and control proceeds to 350. 
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If there is no request to stop capture, then at 345 it is tested whether there 
is a request to play. If there is such a request the captured asset is played at 353 and control 
returns for 341. 

If there is no request to play, then, at 346, it is tested whether there is a 
5 timeout. If so, the asset is saved in a file at 347, and a notification message is made for the 
plug-in DLL at 349. Then control passes to 350. 

If there is no timeout after box 346, then an error must have occurred and 
an error message is made for the plug-in DLL at 348 and control passes to box 350. 
At 350, the asset capture DLL is deinitialized. 
10 Returning to Fig. 3a, once the user selection is received, the plug-in 

software is to be retrieved at 305. The system must then determine, at 306, if the user has 
previously captured this type of asset, or more specifically, whether the user already has the 
relevant plug-in installed on local disk. 

If the user needs a plug-in, then at 312, that plug-in is downloaded from 
15 the web server. Once the system has a plug-in, the plug- in needs to be loaded and run at 
307. Captured assets are then received from the user computer at 309 and the assets are 
added to the multimedia message at 310. The assets are transferred to the web server using 
standard 'Multipart forms' format. 

The following is pseudocode for capture and transfer of an asset by the 
20 browser on the user computer. 

Initialize plug-in 

Read parameters passed by browser (e.g. user i.d., session i.d., asset 

type) 

Scan local setup for a device or devices to be used for asset capture 
25 Initialize local devices 

Display user-interface 

Process user input to start, stop, edit capture 
Save captured asset on local storage 
Ask browser to stop connection with server 
30 Use browser to submit multi-part HTML form containing stored asset 

Pass session and user i.d.'s to server 
End 

The following is pseudo code for the server 
If first time user 
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then register user 



10 



15 



20 



else 

check identity 
assign session i.d. 
show screen with asset buttons 
if user clicks on asset button 

then serve empty html file with embedded plug-in object 
else if user clicks on address/send button 

then take user through addressing and sending of message; 
notify recipient 
if browser connects for upload of captured asset 

then save uploaded asset using session i.d. and user i.d. as identifier of 

message being composed 

Fig. 5c shows a screen as seen by the user when running the photo 



scanning plug-in in the preferred embodiment. Fig. 5d shows a screen as seen by the user 
when running the video recording plug-in. Fig. 5e shows a screen as seen by the user when 
running the audio recording plug-in. Fig. 5f shows a part of a finished letter as displayed in 
the preferred embodiment. 



When the recipient is notified of the message, the recipient needs to be 



given some identification of the message. In the case of the above pseudo code, the session 
i.d. and user i.d. are used to identify the message for the recipient. This serves as a kind of 
security, since only the recipient will know these i.d.'s. If additional security is desired, 
password access can be established for the messages or the messages can be scrambled. 



Fig. 4 shows more detail of the configuration of the software modules on 



the user system. The web server 102 is the same as in Fig. 2. The Figure also shows the 
web browser 401, the plug-in 402, and the asset capture device 403. 
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CLAIMS: 



1 . A server comprising 

interface means for interfacing to a network and for receiving therefrom and 
providing thereto multimedia information; 

memory means; 

5 processing means, cooperating with the interface means and the memory means, 

for 

organizing the multimedia information in downloadable or browsable 
format in the memory means for access by a remote receiver via the interface 
means, which multimedia information includes multimedia e-mails with personal 
10 content, which multimedia e-mails are received via the interface means from at 

least one remote sender; 

coordinating message identification information received via the interface 
from the remote receiver; 

allowing the remote receiver to access a particular multimedia e-mail. 
15 2. A system comprising 

a network; 

the server of claim 1; and 

the remote sender, comprising a multimedia e-mail composition facility. 

3. The system of claim 2, wherein the multimedia e-mail composition facility 
20 comprises at least one plug-in or plug-in equivalent downloaded from the server. 

4. The system of claim 3 wherein the remote receiver is able to browse the 
particular multimedia e-mail, but not able to receive multimedia e-mail sent over the 
network. 

5. A method of multimedia communication comprising the steps of 

25 first receiving, via a network, at a server, a multimedia e-mail composed by a 

remote sender; 

organizing the multimedia e-mail in a database for later retrieval; 
sending information, via the network, about the stored e-mail to a remote 
receiver from the remote sender; 
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second receiving, via the network, at the server, a request for the multimedia e- 
mail from the remote receiver; and 

allowing, by the server, the remote receiver to access the multimedia e-mail via 

the network. 

5 6. The method of claim 5 wherein the remote sender is unable to send the e- 

mail directly to the remote receiver via the network. 

7. The method of claim 5 wherein the server provides a home page for 
browsing via the world wide web and the server requires the remote receiver or the remote 
sender to browse at least part of the home page prior to the first or second receiving step, 

10 respectively. 

8. A method of composing a multimedia e-mail at a send station, the method 
comprising the following steps; 

a. accessing a server via a network; 

b. displaying screen information received from the server indicating user 

15 choices; 

c. transmitting asset selection information from the send station to the server; 

d. determining whether the send station has a plug-in or plug-in equivalent for 
acquiring a desired asset; 

e. upon a negative result of the determining step, downloading the plug-in or 
20 plug-in equivalent from the server for acquiring the desired asset; and 

f. acquiring the desired asset using the plug-in or plug-in equivalent. 

9. A method for using a server to assist in composing a multi-media e-mail, 
the method comprising the steps of: 

a. receiving an access request from a send station; 
25 b. transmitting screen information indicating user choices to the send station; 

c. receiving asset selection information from the send station; 

d. determining whether the send station has a plug-in or plug-in equivalent for 
acquiring a desired asset; 

e. upon a negative result of the determining step, downloading the plug-in or 
30 plug-in equivalent to the send station for acquiring the desired asset; and 

f . receiving the desired asset from the send station after the send station 
acquires the desired asset using the plug-in or plug-in equivalent. 

10. The method of claim 9 further comprising the steps of 
using the desired asset to compose a multimedia e-mail; and 
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retaining the multimedia e-mail in browsable format on the server. 




FIG. 4 




FIG. 5B 



